Advanced Content and Interface Personalization through Conversational Behavior and Affective Embodied Conversational Agents

Conversation is becoming one of the key interaction modes in HMI. As a result, the conversational agents (CAs) have become an important tool in various everyday scenarios. From Apple and Microsoft to Amazon, Google, and Facebook, all have adapted their own variations of CAs. The CAs range from chatbots and 2D, carton-like implementations of talking heads to fully articulated embodied conversational agents performing interaction in various concepts. Recent studies in the field of face-to-face conversation show that the most natural way to implement interaction is through synchronized verbal and co-verbal signals (gestures and expressions). Namely, co-verbal behavior represents a major source of discourse cohesion. It regulates communicative relationships and may support or even replace verbal counterparts. It effectively retains semantics of the information and gives a certain degree of clarity in the discourse. In this chapter, we will represent a model of generation and realization of more natural machine-generated output

Neprototipni diskurzni označevalec zdaj

Prispevek predstavlja kvalitativno študijo primera diskurznega označevalca zdaj, ki je bil doslej proučevan v prozodično nevpeti strukturi oz. kot prototipni diskurzni označevalec. Strukturna, tj. prozodična, analiza kaže, da se diskurzni označevalec zdaj pojavlja tudi v neprototipni obliki. Na podlagi semantične analize, ki jo podpremo s testom parafraziranja in prevajanja, ugotavljamo, da neprototipni zdaj uvaja manjše tematske premike kot njegova prototipna različica in se pojavlja predvsem v vlogi poudarjanja, medtem ko je za prototipni zdaj značilna predvsem vloga strukturiranja diskurza. The present paper presents a qualitative case study of the Slovene discourse marker zdaj (now), which, thus far, has been explored in its prosodically prominent structure or as a prototypical discourse marker. The structural, i.e., the prosodical analysis, highlights that the discourse marker zdaj also manifests in its non-prototypical form. By means of semantic analysis, underpinned by p araphrase a nd t ranslation t ests, we s uggest t hat t he n on-prototypical form signals subtler frame shifts than its prototypical counterpart, and that it primarily functions as an emphasis device, whereas the prototypical zdaj predominately performs discourse structuring functions

Multilingual Chatbots to Collect Patient-Reported Outcomes

With spoken language interfaces, chatbots, and enablers, the conversational intelligence became an emerging field of research in man-machine interfaces in several target domains. In this paper, we introduce the multilingual conversational chatbot platform that integrates Open Health Connect platform and mHealth application together with multimodal services in order to deliver advanced 3D embodied conversational agents. The platform enables novel human-machine interaction with the cancer survivors in six different languages. The platform also integrates patients’ reported information as patients gather health data into digital clinical records. Further, the conversational agents have the potential to play a significant role in healthcare, from assistants during clinical consultations, to supporting positive behavior changes, or as assistants in living environments helping with daily tasks and activities

Can Turn-Taking Highlight the Nature of Non-Verbal Behavior: A Case Study

The present research explores non-verbal behavior that accompanies the management of turns in naturally occurring conversations. To analyze turn management, we implemented the ISO 24617-2 multidimensional dialog act annotation scheme. The classification of the communicative intent of non-verbal behavior was performed with the annotation scheme for spontaneous authentic communication called the EVA annotation scheme. Both dialog acts and non-verbal communicative intent were observed according to their underlying nature and information exchange channel. Both concepts were divided into foreground and background expressions. We hypothesize that turn management dialog acts, being a background expression, co-occur with communication regulators, a class of non-verbal communicative intent, which are also of background nature. Our case analysis confirms this hypothesis. Furthermore, it reveals that another group of non-verbal communicative intent, the deictics, also often accompany turn management dialog acts. As deictics can be both foreground and background expressions, the premise that background non-verbal communicative intent is interlinked with background dialog acts is upheld. And when deictics were perceived as part of the foreground they co-occurred with foreground dialog acts. Therefore, dialog acts and non-verbal communicative intent share the same underlying nature, which implies a duality of the two concepts

PALANTIR: Zero-trust architecture for Managed Security Service Provider

The H2020 PALANTIR project aims at delivering a Security-as-a-Service solution to SMEs and microenterprises via the exploitation of containerised Network Functions. However, these functions are conceived by third-party developers and can also be deployed in untrustworthy virtualisation layers, depending on the subscribed delivery model. Therefore, they cannot be trusted and require a stringent monitoring to ensure their harmlessness, as well as adequate measures to remediate any nefarious activities. This paper justifies, details and evaluates a Zero-Trust architecture supporting PALANTIR’s solution. Specifically, PALANTIR periodically attests the service and infrastructure’s components for signs of compromise by implementing the Trusted Computing paradigm. Verification addresses the firmware, OS and software using UEFI measured boot and Linux Integrity Measurement Architecture, extended to support containerised application attestation. Mitigation actions are supervised by the Recovery Service and the Security Orchestrator based on OSM to, respectively, determine the adequate remediation actions from a recovery policy and enforce them down to the lower layers of the infrastructure through local authenticated enablers. We detail an implementation prototype serving a baseline for quantitative evaluation of our work

PALANTIR demo: leveraging SecaaS model for managing threats in industrial environments

PALANTIR, an EU-funded Innovation Action project, delivers tailored and pervasive resource protection through the Security-as-a-Service paradigm. This demo presents the architecture and validates the prototype against three threat scenarios with botnet, data breach and tampering attacks

An expressive conversational-behavior generation model for advanced interaction within multimodal user interfaces

Doktorska disertacija obravnava problematiko načrtovanja, oblikovanja in samodejnega generiranja koverbalnega obnašanja na poljubnem besedilu, ki je hkrati tudi sinhrono z generiranim govornim signalom. Cilj predstavljene disertacije je oblikovati in zasnovati fleksibilen in učinkovit sistem, ki glede na poljubno vhodno besedilo, semiotično slovnico, leksikon in slovar oblik načrtuje, oblikuje in realizira premike delov telesa (roke, dlani, prsti, glava in ustnice), ki so tako pomensko kot prozodično ustrezno sinhronizirani s spremljajočim govorom. Takšnemu obnašanju pravimo koverbalno obnašanje (oz. pogovorno obnašanje) in ga lahko uporabljamo v multimodalnih vmesnikih različnih sistemov in domen uporabe. Večina današnjih sistemov s pogovornimi agenti, ki pogovorno obnašanje predstavljajo v obliki koverbalnih gibov, temelji na semantičnem in/ali sintaktičnem povezovanju besedila in pogovornega obnašanja, načrtovanju obnašanja s stališča funkcionalnega namena ali sinhronizaciji in realizaciji pogovornega obnašanja, opisanega v obliki proceduralnega opisa. Pogosto arhitekture sistemov, ki integrirajo pogovorne agente, uporabljajo tristopenjsko definicijo pogovornega obnašanja, in sicer: načrtovanje namena, načrtovanje obnašanja in realizacijo obnašanja. Ključni problem samodejnega generiranja naravnega koverbalnega obnašanja predstavljata predvsem naslednji stopnji: načrtovanje namena in načrtovanje obnašanja. V pogovornem obnašanju slovnice ni mogoče definirati kot skupek uniformnih pravil. Posledično torej ni moč definirati končne množice pravil, po katerih bi obnašanje in namen lahko deterministično načrtovali. Vendar pa različne analize in označevanje pogovornega obnašanja kažejo na visoko semantično korelacijo koverbalnega obnašanja in govora ter na njuno prozodično skladnost. Vendar je semantični prostor ogromen in ga v celoti praktično ni mogoče opisati. Zaradi tega problem pogovornega obnašanja največkrat delimo na podprobleme, kot sta funkcionalnost koverbalnosti ali parametrizacija semantičnega prostora. Tako dobimo manjše in bolj obvladljive podprostore, v katerih lažje definiramo semantične relacije med koverbalnim in verbalnim obnašanjem. V obravnavani doktorski disertaciji proces tvorjenja pogovornega obnašanja predstavimo v obliki gest kot povezave kontekstno neodvisnih motoričnih sposobnosti (oblik, poz in gibov, ki jih pogovorni agent lahko prikaže) ter namena (konteksta) uporabe motoričnih sposobnosti. Namen besedila in posledično spremljajočega koverbalnega obnašanja definiramo na osnovi modularnih jezikovno odvisnih slovnic, ki temeljijo na razčlenjevanju in klasifikaciji besedila v semiotične razrede. V tej točki poleg obstoječih raziskav s področja povezave semiotičnih razredov in govora oblikujemo in definiramo novo označevalno shemo, ki vključuje tako funkcionalni, kontekstno odvisni nivo povezave govora in gest kot kontekstno neodvisen opisni nivo, ki omogoča zajemanje poz in oblik na visoki ločljivosti, in lastni multimodalni korpus pogovornega obnašanja, korpus EVA. Rezultate analize in označevanj uporabimo za tvorjenje semiotične slovnice, slovarja oblik in leksikona gest. Predlagani model ekspresivnega pogovornega obnašanja formulira različne koverbalne oblike (geste) na neoznačenem poljubnem besedilu. Predlagani model temelji na analizi in opisovanju spontanega pogovornega dialoga v obliki ekspresivne anotacijske sheme (združuje in povezuje tako funkcionalno kot oblikovno anotacijo) in vsebuje dovolj bogat slovar kontekstno neodvisnih oblik in gibov, s katerimi lahko predstavimo kompleksne motorične sposobnosti agenta in zmožnost prilagoditve gibov različnim kontekstom uporabe (slovnicam) ter imitacije emocionalnih stanj in emocionalnega govora.The presented thesis deals with the problems of planning, design, and automatic generation of coverbal behavior, in relation to general input text, which is aligned with synthesized speech. The aim of the thesis is to develop and design a flexible and efficient system which is (with respect to an arbitrary input text, semiotic grammar, lexicon and dictionary of conversational shapes), capable of planning, generating and realizing conversational behavior in form of movement of body parts (hands, arms, fingers, head and lips) that are both meaningfully, and in terms of prosody synchronized with accompanying speech. This type of behavior is then called co-verbal behavior, and can be used in multimodal interfaces for different systems. The key problem, when automatically generating natural co-verbal behavior from arbitrary text, arises mostly from the intent planning, and behavior planning stages. Namely, the conversational behavior does not comply with an underlying uniform grammar, therefore, a finite set of rules for deterministic planning of the behavior and purpose does not exists. However, several studies and annotations show that a high semantic correlation of co-verbal behavior and speech is present, together with essential prosodic coherence. However, the semantic space in which gestures must be coupled with different intents is enormous and virtually impossible to be described. Therefore, the intent planning and behavior planning are often divided into sub-problems, where behavior generation is based on communicative functions (e.g. feed-back). The generation of gestures is based on speech prosody and limited semantic grammar (gesticon), or the semiotic parameterization of semantic space (e.g. generation of iconic outlines, generation of deictic signals). These approaches define smaller and far more manageable sub-sets, in which the specific semantic/synthetic and prosodic relationships between co-verbal and speech behavior are easily found. The proposed expressive conversational behavior generation model formulates various forms of coverbal behavior (gestures) with respect to arbitrary and un-annotated text. It is based on the analysis and annotation of spontaneous dialog dialogue and contains rich context-independent shape and movement repository, which can represent complex motor skills, the ability to adjust the agent movement to different contexts of use (grammar), and to imitate emotional states and emotional speech. The ability to re-use those shapes and movements, maintained within the dictionary and the reduction of the semantic dependency into semiotic classes that depend on morphology, allows for the conversational agent to express a wide range of communicative concepts (such as: diversity, motivation, relevance, relation to theme, etc.). The use of these agents can be extended in a variety of contexts within multimodal interfaces, such as: web readers, including RSS content, storytellers, intelligent virtual managers, IPTV environments, virtual companions, listeners and therapists, etc. The conducted perception experiments show that by using the proposed expressive conversational behavior generation model adequate degree of naturalness maybe achieved